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ABSTRACT: Computational theories of structurc-from-motion and stereo vision only specify 
the computation of three-dimensional surface information at points in the image at which the ir- 
radiancc changes — for example, the zero-crossings of a V 2 G operator applied to the image. Yet, the 
visual perception is clearly of complete surfaces, and this perception is consistent for different observ- 
ers. Since mathematically the class of surfaces which could pass through the known boundary points 
f*^. provided by the stereo system is infinite and contains widely varying surfaces, the visual system must 

incorporate some additional constraints besides the known points in order to compute the complete 
surface. 

Using die image irradiance equation, we derive the surface consistency constraint, informally 
referred to as no news is good news. The constraint implies that the surface must agree with the 
information from stereo or motion correspondence, and not vary radically between diese points. An 
explicit form of this surface consistency constraint is derived, by relating the probability of a zero- 
crossing in a region of the image to die variation in die local surface orientation of the surface, 
provided that the surface albedo and die illumination are roughly constant. 

The surface consistency constraint can be used to derive an algorithm for reconstructing the 
surface that "best" fits die surface information provided by stereo or motion correspondence. 



This report describes research done at the Artificial Intelligence Laboratory of the Massachusetts 
Institute of Technology. Support for die laboratory's artificial intelligence research is provided in 
part by the Advanced Research Projects Agency of the Department of Defense under. Office of 
Naval Research contract N00014-80-O0505 and in part by National Science Foundation Grant 79- 

23110MCS. fi MASSACHUSETTS INSTITUTE OF TECHNOLQGX Ml 



i r ' 



<«**■"% 



1. Introduction 

Although our world has three spatial dimensions, the projection of light rays onto the retina 
presents our visual system with an image of the world that is inherently two-dimensional. We must 
use such images to physically interact with this three-dimensional world, even in situations new to 
us, or with objects unknown to us. That we easily do so implies that one of the functions of the 
human visual system is to reconstruct a three-dimensional representation of the world from its two- 
dimensional projection onto our eyes. 

Methods that could be used to effect this three-dimensional reconstruction include stereo vision 
[e.g. Wheatstone, 1838; Helmholtz, 1925; Julcsz, 1971] and structure from motion [for example, Miles, 
1931; Wallach and O'ConnclI, 1953; Johannson, 1964]. Both of these methods may be considered as 
correspondence techniques, since they rely on establishing a correspondence between identical items 
in different images, and using the difference in projection of these items to determine surface shape. 

Most of the current computational theories of these processes [Marr and Poggio, 1979; Crimson, 
1980, 1981a; Mayhcw and Frisby, 1981; Ullman, 1979; Longuct-Higgins and Prazdny, 1980] argue 
that the correspondence process cannot take place at all points in an image. Rather, the first stage 
of the correspondence process is to derive a symbolic description of points in the image at which the 
irradiance undergoes a significant change [Marr and Hildrcth, 1980]. This symbolic representation 
(called the primal sketch [Marr, 1976; Marr and Hildrcth, 1980]) forms the input to die second stage 
of the process in which the actual correspondence is computed. As a consequence of the form of the 
input, the correspondence process can compute explicit surface information only at scattered points 
in the image. Yet our perception is clearly of complete surfaces. For example, in Figure 1, a sparse 
random dot stereogram yields the vivid perception of a square floating in space above a background 
plane, rather than a collection of dots suspended in space. This suggests that some type of filling-in, or 
interpolation of surface information, is taking place in the visual system. 

This poses an interesting problem, since mathematically the class of surfaces that could pass 
through the known boundary points provided by die stereo algorithm is infinite and contains widely 
varying surfaces. The implication of our perception of complete surfaces, which is consistent for 
different viewers, is that the visual system must incorporate some additional constraints, besides the 
known points, in order to compute the complete surface. 
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THE COMPUTATIONAL CONSTRAINT 
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Figure 1. A Sparse Random Dot Pattern. Although the density of dots is very small, the perception 
obtained upon fusing this pattern is one of two disjoint planes, rather than dots isolated in depth. 
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In this paper, wc will use the image irradiance equation [Horn, 1977] to derive the surface 
consistency constraint, informally referred to as no news is good news. The constraint implies that the 
surface must agree with the information from stereo or motion correspondence, and not vary radically 
between these points. We will derive an explicit form of this surface consistency constraint, by relating 
the probability of an abaipt irradiance change in a region of the image to the variation in the local 
surface orientation of the corresponding surface. Wc will then indicate how tills constraint can be 
used to derive an algorithm for reconstructing the surface that "best" fits the surface information 
provided by stereo or motion correspondence [Grimson, 1981b, 1982]. 

2. The Computational Constraint 
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The input representation used by the stereo algorithm [Marr and Poggio, 1979; Grimson, 1981a, 
1981b] consists of a symbolic representation of points at which the image irradiance undergoes a 
significant change. These points are identified by the zero-crossings of a Laplacian of a Gaussian 
(V 2 G) operator applied to the image; that is, the image is convolved with a V 2 G operator and the 
points at which the resulting convolution changes sign are located [Marr and Hildreth, 1980; Hildreth, 
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1980]. As a consequence of the correspondence process, explicit three-dimensional information about 
the surface shape will be computed only at these zero-crossings. 

Suppose one were to attempt to construct a complete surface description based only on the 
surface information known along the /cro-crossings (where by complete, we mean that an explicit 
surface value is assigned to each point on a rectangular grid). An infinite number of surfaces would 
consistently fit the boundary conditions provided by these surface values. Yet there must be some 
way of deciding which surface, or at least which small family of surfaces, could give rise to the zero- 
crossing descriptions. This means that there must be some additional information available from the 
visual process which, when taken into account, will identify a class of nearly indistinguishable surfaces 
that represent the visible surfaces of a scene. 

In order to determine what information is available from the visual system, one must first care- 
fully consider die process by which the zero-crossing contours are generated. For instance, sudden 
changes in die reflectance of a surface, caused by surface scratches or texture markings, will give rise 
to zero-crossings in the convolved image [Marr and Hildreth, 1980; Hildreth, 1980]. Of more interest 
here is the fact that sudden or sharp changes in die shape of die surface will under most circumstances 
also give rise to zero-crossings. This can be used to constrain die possible shapes of surfaces that could 
produce particular surface values along die zero-crossing contours 

We illustrate die basic argument with an example. Suppose one is given a closed zero-crossing 
contour, within which there are no other zero-crossings. An example would be a circular contour, 
along which die depth is constant. There are many surfaces which could fit this set of boundary 
conditions. One such surface is a flat disk. One could, however, also fit other smooth surfaces to this 
set of boundary conditions. For example, the highly convoluted surface formed by sin (a; 2 -f y 2 )* 
would be consistent with die known disparity values (see Figure 2). Yet in principle, such a rapidly 
varying surface should give rise to other zero-crossings. This follows from the observation diat if the 
surface orientation undergoes a periodic variation, then it is likely that the irradiance values will also 
undergo such a variation. Since the only zero-crossings lie at die borders of the object, diis implies 
diat the surface sin (x 2 -f y 2 ) 2 is a highly improbable representative surface for this set of boundary 
conditions. 

Hence, die hypodiesis, which will be evaluated in die following sections, is diat the set of zero- 
^0*^ crossing contours contains implicit information about die surface as well as explicit information. If 
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Figure 2. Possible Surfaces Filling Depth Values at Zero-Crossings. Given boundary conditions 
of a circular zero-crossing contour, along which the depth is constant, there are many possible 
surfaces which could lit the known depth points. Two examples are a flat disk, and the highly 

convoluted surface formed by sin {x 2 -f- y 2 ) 5 , shown here. (From Grim son [1981b].) 



one can determine a set of conditions on the surface shape that cause inflections in the irradiance 
values, then one may be able to determine a likely surface structure, given a set of boundary condi- 
tions along the zero-crossing contours. 

2.1 No News is Good News 

In general, any one of a multitude of widely varying surfaces could fit the boundary conditions 
imposed along the zero-crossings. Our intention is to show that to be completely consistent with 
the imaging process, such surfaces must meet both explicit conditions and implicit conditions. The 
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explicit conditions arc given by the depth values along the zero-crossing contours. The implicit condi- 
tions are that the surface must not impose any zero-crossing contours other than those which appear 
in the convolved image. This implicit condition leads to the surface consistency constraint[Gn\mo\\ 
1981b], namely: 

The absence of zero-crossings constrains the possible surface shapes. 

Just as the presence of a zero-crossing tells us that some physical property is changing at a given 
location, the absence of a zero-crossing tells us the opposite, that no physical property is changing, 
and in particular that the surface topography is not changing in a radical manner. We informally 
refer to this constraint as no news is good news, since it says that in general die surface cannot change 
radically without informing us of this fact by means of zero-crossings. Note that in practice, the 
convolved image may not signal the entire extent of a discontinuity in the surface topography, but at 
least some portions of the discontinuity will be evidenced by a zero-crossing. In order to complete 
such a partial discontinuity, a subjective contour must be inserted between die known portions [Brady 
^**s and Grimson, 1981]. 

In order to make explicit any constraints on die shape of die surface for locations in the image 
not associated with a zero-crossing, we will carefully examine die factors which influence irradiance 
and hence image intensity. (Note that while irradiance refers to die illuminant flux emitted from the 
surface of an object and intensity refers to die brightness recorded by a sensor, we will use the terms 
intcrchangably, since they are proportional [Horn and Sjoberg, 1979].) These factors will be expressed 
in die image irradiance equation, which describes die manner in which a particular irradiance is 
formed at a point in the image. Two goals are kept in mind. The first is to determine what surface 
conditions will cause a local change in irradiance, and the second is to combine this constraint with 
die input from the visual processes, such as stereo or structure-from-motion, in order to design an 
algorithm for interpolating surface information. 

The basic result, corresponding to the intuitive argument given above, is that the probability of a 
zero-crossing occurring, in regions where the illumination is roughly constant and die surface material 
does not change, is a monotonic function of die variation in the orientation of the surface normal. 
This means diat the probability of a zero-crossing increases as the variation in surface orientation 
increases. By inverting this argument, we will show that the best surface to fit die known data is that 
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IMAGE FORMATION 6 

which minimizes the variation in surface orientation since this surface is most consistent with the zero- 
crossings in the convolved image. 



3. Image Formation 

In order to examine the process of zero-crossing formation, a review of the factors involved in 
the formation of an image will be presented. One factor is the geometry of the projection (cither 
orthographic or perspective) from die scene to the image. The second factor, which we will review 
briefly below, is the manner. in which image irradiancc values at a point are formed. A detailed 
analysis of these processes has been undertaken by several investigators [Horn, 1970, 1975, 1977; 
Nicodcmus, ct al., 1977; Woodham, 1978; Horn and Sjoberg, 1979]. 

3.1 Grey-Level Formation 

/■-s In this section, wc will outline the factors involved in die creation of image irradiances in order 

to illustrate the relationship between changes in irradiance and zero-crossings in die convolved image, 
(most of the discussion is based on Horn and Sjoberg, [1979]). To do this, wc need to determine what 
irradiance value will be associated with a particular image location. 

Hie apparent "brightness*' recorded by an imaging device is a measurement of image irradiance 
E n the radiant flux striking a unit area of the receptive field. The flux reaching a small portion of 
the receptive field will be exclusively a function of a corresponding small surface element on some 
object in the scene, provided die imaging system is properly focused and the lens is small relative to 
its distance from die object [Horn and Sjoberg, 1979]. We want to specify the factors that determine 
the image irradiance at a point in the image, as a function of die corresponding surface element. 

In general, there are diree factors governing the irradiance recorded at any position in the 
image: 

(1) the amount of radiant flux striking a surface, a function of die distance of die surface from the 
source r, the orientation of the surface relative to the source, and the intensity I of the source 
itself; 

(2) the percentage of incident flux reflected by die surface (as opposed to the percentage absorbed 
or transmitted), a function of the surface material and usually described by the albedo p\ and 
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(3) the distribution of that reflected flux as a function of direction, usually described by a reflectance 
function. 

We shall show that a sharp change in each of these factors will give rise to a zero-crossing in the con- 
volved image. Further, in regions where both die incident illumination and absorption characteristics 
of the surface material are roughly constant, an equation relating die recorded irradianccs to the shape 
of die surface will be derived. This image irradiance equation will be used in later sections to verify 
our surface consistency constraint. 

Most surfaces have the property that the reflectance is not changed by rotating a surface element 
about its normal (exceptions arc diffraction gratings, irridescent plumage and "tiger's eye"). Such 
surfaces are referred to as isotropic. If a surface possesses this property, only Uiree angles are needed 
to determine reflectance, as shown in Figure 3. The angle between the local surface normal and the 
incident ray is called die incident angle and is denoted by z. The angle between die local surface 
normal and die emitted ray is called the view angle and is denoted by e. The angle between the 
incident and emitted rays is called the phase angle and is denoted by g. In the following, we shall 
assume that the surfaces are isotropic. This reduces die image irradiance to a function of/, r, i, e } g 
and p, where / is die intensity of the source, r is the distance of die source from the surface, i, c, and g 
are the angles specified above, and p is the albedo of die surface material. 

We know that the incident flux density follows an inverse-square reduction as a function of dis- 
tance. If two surfaces lie at very different depdis, and all other factors are roughly equal, this inverse- 
square dependence will cause a noticeable difference in the irradianccs associated with die different 
surfaces. If die projections of the two surfaces are adjacent in the image, the different irradianccs 
will give rise to a zero-crossing in die convolved image. This is to be expected, since the Marr- 
Hildreth theory of edge detection was based on the requirement of detecting changes in the image 
corresponding to changes in some physical property of the surfaces. In this case, the changing physical 
property is object continuity. 

Given that these situations will cause a zero-crossing, we can restrict our attention to regions of 
the image between the borders of the objects. We will further assume that die source is distant relative 
to die surfaces. This has two consequences. The first is that each surface receives roughly uniform 
illumination. This implies that the I/r 2 term is roughly constant and may be ignored in what follows. 
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GREY-LEVEL FORMATION 



SOURCE 




Figure 3. The Imaging Geometry. The incident angle i is the angle between the incident ray 
and surface normal. The view angle e is the angle between the emergent ray and the surface 
normal The phase angle g is the angle between the incident and emergent rays. (Redrawn from 
Woodham, [J 978]). 



The second is that the angle between die viewer direction and due source direction (the phase angle g) 
will be roughly constant over each surface. ' 

If die surface material changes, there will frequently be an associated change in the albedo. If 
all other factors are roughly equal, tliis will cause a change in the irradiances and there will be a 
corresponding zero-crossing in the convolved image. This is again expected from the Marr-Hildreth 
theory of edge detection: a changing physical property of the surface should correspond to a zero- 
crossing in the convolved image. In general we shall restrict our attention to regions of the surfaces 
between such material changes, so that the albedo p is roughly constant. In these regions, we can 
consider the image irradiances to be a function of i, e and p (ignoring scale constants). 

If we assume dial die source is distant enough and small enough to be treated as a point source, 
then we can simplify die geometry further by reducing the angles i and e to a specification of die local 
surface normal, in the following manner. Consider a Cartesian coordinate system with z-axis aligned 
with the line of sight, and s-axis some arbitrary direction in the plane normal to the line of sight. 
Suppose die surface is specified in Lie coordinate system by z = f{x t y). Elementary calculus states 
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that the vectors 

are tangent to die surface at a given point. The cross product 
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\dx '&y' J 



is normal to die tangent plane, and hence to the surface. Defining 

— *L —®± 

P ~ dx q ~ dy 

the normal vector becomes {p, q, — 1}. The quantity (p, g) is usually referred to as the gradient and 
the space of all points (p, 17) is referred to as gradient space [Huffman, 1971; Mackworth, 1973]. 

In diis case, die angles i and e can be straightforwardly transformed into die normal components 
p and q [Horn, 1977]. Since we are considering die case of a distant point source, its direction reladve 
to die surface can be represented by a direction vector {p S) q Hf —l}.\ straightforward calculation of 
the vector dot products shows that the angles i and e are given by: 

cos(i)= 1+Pft + TOi _ 

cos(e) = 



v /l +p 2 +g 2 

Extended sources (sources whose size relative to their separation from the scene cannot be treated 
as a point source, for example, fluorescent lights in a normal room) can be frequently treated by su- 
perposition of point sources, or as being equivalent to an appropriately chosen point source [Brooks, 
1978; Silver, 1980]. Thus, the irradiancc reduces to a function of p, q and p. 

Finally, if we assume that die objects are distant reladve to the viewer, the image projection may 
be treated as orthographic. In diis case, the coordinate system of the image and the coordinate system 
of the scene can be treated as identical so that we obtain the image irradiance equation: 

E{x, y) = p{x, y)R{ p(z, y), q(x, y) ) 

where E(x, y) is the image irradiance recorded at a point in the image, p is the albedo associated 
with die surface intersecting die ray from (x } y) and R(p, q) is die reflectance map [Horn, 1970, 1975, 
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THE INTERPOLATION THEOREM 10 

Horn and Sjobcrg, 1979, Horn and Bachman, 1977]. Note that tlic reflectance map R differs from 
the reflectance function for a surface [Nicodernus, et al., 1977]. The reflectance function describes 
the amount of flux reflected in a particular direction, given an incident flux from a second given 
direction. The reflectance map combines the effects of the reflectance function in the case of uniform 
illumination and isotropic surface material with the illumination geometry and die image sensor's 
view point. In the case of constant albedo the image irradiancc equation reduces to the partial 
differential equation [Horn, 1970, 1975; Horn and Sjobcrg, 1979]: 

E(x, y) = R( p{x } y), q{x, y) ). 

These two equations describe the manner in which an image irradiancc (or grey-level) is obtained at a 
particular point in the image. 

4. The Interpolation Theorem 

We have seen that zero-crossings can arise from many factors. We shall restrict our attention to 
regions in which illumination is constant, albedo is roughly constant and surface material is isotropic. 
Our intention is to show that in such cases, if the surface topography changes radically, the image 
irradiances must also change radically. We will then be able to use die contrapositive statement — if 
the image irradiances do not change radically, then die surface topography also does not change. Note 
that if the albedo or the illumination is not constant over the region, we cannot make this statement. 
That is, there could be situations in which die surface topography does change radically without 
causing a corresponding change in the image irradiances, because one of the other factors, such as 
the albedo, is also changing in such a manner as to mask out die effects of die change in topography. 
These situations are fortunately very rare, since they require a precise meshing of the effects of die 
changing albedo with die changing surface topography and in general are highly dependent on the 
viewer position and illuminant direction. Thus, a small change in the viewer posidon will usually 
decouple die conflicting effects of albedo and surface shape, and the change in topography will give 
rise to a zero-crossing. (In other words, it is generally difficult to paint a radically curved surface so 
as to appear flat from all viewpoints. The opposite is not true, of course. One can easily paint a 
flat surface to appear curved. In other words, we prefer to see topographic changes, as evidenced by 
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11 

the informal observation that photographs with smoothly changing albedo are usually interpreted as 
having smoothly changing topography. } 

The concern here is whether there can be zero-crossings corresponding to topographic rather 
than photometric changes in an object, for example, in the radial sine surface of Figure 2. Indeed, 
if die albedo is roughly constant, the form of the equations indicates that there may be topographic 
effects that could also cause sharp changes in irradiance. The surface conditions under which such a 
change in irradiance can occur are important, since the absence of zero-crossings in a region would 
then imply the absence of such surface conditions for that region. It is precisely these restrictions 
on surface shape which will allow us to determine a surface consistent with the depth values along 
the zero-crossing contours. The basic problem is, under what conditions docs bending of die surface 
force an inflection in the irradiance array? This question will be answered in the following sections by 
considering specific cases. 

There are two points to note, before beginning the mathematical details. The first concerns the 
role of the Gaussian filter in constructing the primal sketch representations upon which the stereo 
correspondence is computed. The Gaussian performs a smoothing of the image, thereby isolating ir- 
radiance changes at a particular scale. At the same time, it removes some of the noise problems which 
arise from using a discrete grid to represent the image. In what follows, we shall concentrate on the 
differential operator (V 2 ) and its role in the creation of die zero-crossings. The effect of the Gaussian 
will not be considered in any detail. He.nce, the surface reconstruction will not account for minor 
surface fluctuations on the scale of the grid resolution. This is not a major problem. (Equivalently, 
the problem may be considered as one of applying die Laplacian V 2 to die image G * E. The 
reconstruction will be based on this image information.) 

The second point concerns the use of the Laplacian operator V 2 as opposed to directional 
derivatives. The mathematical arguments which follow are based on the consideration of zero- 
crossings of the convolution of a directional second-order differential operator with the image, (v • 
V) * E. We have already seen that the Marr-Hildrcth theory of edge detection is based on the use 
of zero-crossings of the image convolved with the Laplacian, V 2 * E. This difference is not critical, 
for die following two reasons. Marr and Hildreth show diat under some simple assumptions, the 
zero-crossings obtained by eidier operator are identical. In particular, the condition of linear variation 
^-\ [Marr and Hildreth, 1980] states that: the irradiance variation near and parallel to a line of zero- 
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GENERAL ARGUMENT 12 

crossings should locally be linear. If this condition is assumed to be true then it can be shown [Marr 
and Hildreth, 1980, Appendix A] that the zero-crossings obtained with the Laplacian are identical to 
those obtained using directional derivatives. Thus, in those situations for which the condition of linear 
variation is valid,. no difference is obtained. In the case where the condition of linear variation docs 
not hold, the difference in the zero-crossings of the operators lies in their position, not in the existence 
of such zero-crossings. For example, at a corner (which frequently carries strong visual information), 
both V 2 * E and (v • V) * E will give rise to zero-crossings. The only difference will be in the exact 
position of the zero-crossings. We shall sec that such variation in position is of negligible consequence 
to the reconstruction process. Hence, it will be assumed that the arguments developed in the next 
section, based on the zero-crossings of a directional second derivative will also generally apply to the 
zero-crossings of die Laplacian. 

Finally, in what follows, the functions p, R and / arc assumed to have continuous second order 
partial derivatives. Throughout, it is assumed that the albedo effects can be ignored relative to the 
topographic ones, that is, the albedo factor may be considered roughly constant and will therefore not 
affect any study of derivatives. Without loss of generality, one may assume that p — 1 and ignore the 
albedo in what follows. 

4.1 General Argument 

The basic hypothesis is that in order for a surface to be consistent with a given set of zero- 
crossings, not only must it give rise to a zero-crossing in the convolved irradiances at those points, 
but it must also not give rise to zero-crossings anywhere else. Under most situations, this restriction 
would require that the surface not change in a radical manner between zero-crossing contours. (For 
example, the surface shown in Figure 2 is not consistent with the boundary conditions of a circular 
zero-crossing contour of uniform depth). It is difficult to prove this assertion in general, since the 
image formation equation includes terms dependent on the imaging process and on the light source 
geometry, as well as factors dependent on the photometric properties of the surface itself. However, 
under some fairly weak assumptions concerning the relative strengths of photometric and topographic 
changes, and the form of the reflectance function, we will prove the surface consistency theorem, 
describing the probability of a zero-crossing as a function of the shape of the surface. The importance 
of this theorem is that it leads to a method for measuring the probability of a particular surface being 
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Figure 4. An Example of a Developable Surface. The component of the surface orientation in 
the y direction is constant for this region of the surface, so that the only variations in surface 
orientation take place in the x direction. 
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inconsistent with the zero-crossing information. This in turn suggests diat it will be possible to derive 
a method for determining the best possible surface to fit the known information [Crimson, 1982]. 

4.2 A One-Dimensional Example 

To illustrate the scope of the surface consistency theorem, we shall consider first the one- 
dimensional case of a developable surface, before proving the general theorem concerning arbitrary 
surfaces. Note that the Laplacian is orientation independent, so that without loss of generality, one 
may rotate the coordinate system of the image to suit our needs. One may assume that die surface has 
the form f(x, y) such that f y (z, y) = q(x, y) = c, in the local region under consideration. Hence, 
the partial derivatives of q vanish, as do any partial derivatives of p involving y. A sample surface is 
shown in Figure 4. 

Suppose that a one-dimensional slice in die x direction of the surface contains at least two 
inflection points. Figure 5 indicates a sample surface and its derivatives. Since q is assumed constant, 
the derivatives of the image irradiate equation are given by: 
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V 2 £ = (v 2 p yi + 2p x R pPx + P (r pp pI + i? pP:r A 

(Note that except where explicitly suited otherwise, we shall use a subscript to denote a partial deriva- 
tive, so that, for example, p x = dp/dx.) In this situation of a developable surface, we can prove the 
following result. 

Theorem 0: Consider a portion of a second differentiate, developable surface oriented along the y 
axis such thatf y {x,y) = q = c, for some constant c. If the following conditions are true: 
L The surface portion contains exactly two inflection points in the x direction, at x\ andx 2} 
2. At the points x\ and x% normalized changes in albedo are dominated by normalized changes in 
reflectance, 



V 2 p 

p 


< 


RpPxx 

R 



3. The reflectance map R does not pass through an extremum in (his region of the surface, 

4. The reflectance mapR is not constant over this region of the surface, 

5. The albedo p is non-zero, 

then there exists a point x { < x f < x 2 such that V 2 E{x f ) = 0. That is, there exists a zero- crossing 
here. 

Proof Hie signum function is defined by: 

' 1 if x > 

sgn{x) = < — 1 if x < 

s if x = 

From the derivatives of the image equation, 

V 2 E = (V 2 p]R + 2p x R pPx + p(R pp pl + R pPxx ). 

At the inflection points x x and x 2 , p x {xi) = 0. Hence, evaluation of the equation yields 

V 2 E(xi) = V 2 p(x i )R{p{x l )) + P^RMx^pUxi) fori = 1, 2. 

Condition (2) implies that the albedo changes are negligible in this region, so that the first term may 
be ignored, 

sgn(V 2 E{xi)) = sgn(p{x i )R p {p(x i ))p xx (x i )y 
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Figure 5. One-Dimensional Example, The top figure illustrates a slice of the surface, containing 
two inflection points. The second figure illustrates the first derivative of the surface function. The 
two inflection points of the surface correspond to extrema in the first derivative. The third figure 
illustrates the second derivative of the surface function. The two inflections in the surface correspond 
to zero-crossings in the second derivative. The bottom figure illustrates the third derivative of the 
surface function. Between the points corresponding to the two inflection points of the surface, the 
third derivative contains a zero-crossing. (From Crimson [1981b]). 
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Condition (5) implies that sgn(p) = 1. Observing that 

sgn{xy) = sgn(x) sgnfy), 
the sign of the convolved intensity function at die surface inflection points is given by 

sgn(V 2 E{xi)) = sgn(R p (p{xi))p xx (xi)). 
Condition (3) implies diat R p docs not change sign in this region of the surface and hence 

sgn(R p {p{xi))) = sg7i(R p (p{x 2 ))). 

Note that p x = at x i} x 2 . The fact that there are exactly two inflections on die surface implies that 
p x =£ over die neighborhood (a?i, x 2 ). Thus, 

sgn(p xx {x { )) 7^ sgn(p xx (x2)), 

and thus 

sgn{V 2 E{x l ))^sgn{V 2 E{x 2 )). 

The second differentiability of the surface implies Uiat there exists a point x 1 £ [x it x 2 ) such diat 
V 2 I{x f ) = . a 

The contrapositivc of diis dieorem states something important about the possible surfaces which 
can fit the known dcpdi information. Specifically, given the conditions of die theorem, if there is a 
set of known depdi points to which a surface is to be fit, there cannot be two or more inflections in 
die surface between any two zero-crossing points. If diere were, there would have to be an additional 
zero-crossing there as well. 

Corollary 0.1: Suppose one is given a set of known depth points at a set of zero- crossings, along a 
developable surface. If the albedo p is non-zero and the reflectance map R is not constant for this region 
of the surface, then the surface cannot contain two or more inflection points between any pair of adjacent 
zero-crossings. | 

There are many ways of extending diis result. Some are indicated in the appendix. The most 
important extension is to consider general surfaces, rather than developable ones. We make this 
extension in the following sections. 
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4.3 The Analytic Argument 

Take a planar slice of some surface /, normal to the image plane, along some direction (given for 
example by the angle a between the x axis and the direction of the slice). This is illustrated in Figure 
6. At each point along die resulting curve C\ one may associate a surface orientation, or gradient, 
given by the pair of partial derivatives, p(z, y) = df(x, y)/dx, and q(x, y) = df{x, y)/dy. Thus, one 
may construct a two-dimensional space spanned by a coordinate system with axes given by p and q, 
the gradient space introduced by Huffman [1971] and used by Mackworth [1973], and by Horn [1977] 
to relate the geometry of image projections to die radiometry of image formation. The curve obtained 
by die planar intersection of the surface transforms into a new, parametric curve in gradient space: 

N/(0 = {P(0, 9(0,-1} 

In fact, tliis curve corresponds to die mapping of the normal to the surface, as one moves along the 
planar slice; die subscript/ is used to indicate that this is die normal to die surface /. 

Because the albedo is roughly constant, the image irradianccs arc determined by die reflectance 
map R(p, q). Thus, the irradianccs may be related to Horn's reflectance map, in which one considers 
die surface defined by R(p, q) in gradient space. Hence, the curve C on the original surface will map 
onto a curve N f in gradient space and this may be projected onto the surface R(p, q) to obtain a new 
curve C f (t). At each point t along the parametric curve, die corresponding irradiance is given by 
R (p(t)y 9(0)* (scaled by the constant p 9 which we will ignore without loss of generality). 

We are interested in the conditions on the original surface and the reflectance surface that will 
cause a zero-crossing in die second directional derivative of die image irradiances. (By reflectance 
surface, we mean the reflectance map R(p, q) considered as a surface in gradient space.) Let v be a 
vector in three-space with direction cosines cos a,, cosa y , and cosq^ (the direction cosines refer to 
the cosine of the angle made by the vector and die coordinate axes of die space). Thus, a unit vector in 
the direction of v is denoted by 

y = cosaj-f cosa y j -f- cosa^k 

where i, j, k are unit vectors along the coordinate axes. The directional derivative of $(r), a function of 
a vector r, is the rate of change of* with distance s along die direction v and is denoted by 

(v ♦ V)<l> = cos a,—- + cosa y ~- -f cosa^— . 
ox u dy dz 
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Figure 6. The Generation of Image Irradiance, The curve C is generated by taking the intersection 
of a plane normal to the image plane with the surface. The direction of the plane is given by 
the angle a it makes with the z-axis. By taking the surface normal at each point, this curve C 
can be mapped into gradient space, resulting in a parametric curve N f . Furthermore, this curve 
can be projected onto the reflectance map R (which defines a surface on p-q space) to obtain a 
parametric curve O along which the irradiance is given by die corresponding value of R, (From 
Crimson [1981b]). 
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If die direction vector is constrained to lie in the x-y plane, then cos a z = and cos a y — sin a x . 

Now consider a planar slice normal to the image along die direction v and construct the 
parametric normal curve N f in p — q space. Then the directional derivative of irradiancc along this 
slice is given by 

(vV)£:=i? f ,((vV)p) + /?,((v.V) ? ). 

This can be rewritten as 

(v • V)E = VR • (v • V)N, 

showing that the directional derivative along the slice is given by die projection of the gradient of the 
reflectance surface R onto the tangent of die parametric p-q curve N/. Now, if there are two points 
t u t 2 such that diis dot product vanishes at those points, then Rollc's theorem, and the assumption 
that the surface / is twice continuously differentiate would imply the existence of a zero-crossing in 
the second directional derivative at some point between t\ and i 2 . Thus, to evaluate die probability 
of a zero-crossing in some region of the image, we need to consider die probability of the vectors VR 
and (v • V)N/ being orthogonal. 

There are two cases to consider. The first is when the gradient of-R vanishes at some point, cor- 
responding to an extremum in the reflectance surface. If we assume that over all possible viewpoints, 
VR is uniformly distributed, then die probability of VR = for some point {p{t),q{t)) on the 
parametric curve N/(i) is given by the arclcngth of the curve 
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dt, 
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where the subscript t indicates partial differentiation with respect to die parameter t, 

The second case is when the gradient of R is non-zero and the two vectors are non-trivially 
orthogonal. Consider first die one-dimensional case in which two inflections in a developable surface 
imply the existence of a zero-crossing (as shown in Theorem 0). Since in this case q = c, the 
parametric curve N/(f ) in p-q space is parallel to the p-axis, and reverses direction at each of the 
inflection points. At these points, the curvature of the paramedic curve is essentially infinite, and the 
tangent can be considered to sweep through all possible directions. As a consequence, the tangent 
must be orthogonal to VR for one of those directions, and hence (v • V)E is zero at each point of 
infinite curvature. Thus, Rollers theorem guarantees a zero-crossing along the slice of the image. 
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Now suppose that q j^ c but changes slowly. In this case, the points of infinite curvature 
become points of high curvature. Here, the tangent to the parametric curve sweeps through a large 
range of directions. If we assume that the second order partial derivatives of die reflectance surface R 
arc small, then over a small region of p-q space, VR is nearly constant. Hence, in the region of the 
points of high curvature, die direction of VR is nearly constant, while the direction of the tangent to 
N; sweeps through a large range of directions. Thus, at points of high curvature, the probability of 
(v • V)E — is high. If, on the other hand, the parametric curve is locally a straight line, the tangent 
is constant and the probability of the directional derivative vanishing is small. Thus, along a slice of 
die image, the probability that (v • V)N/ and VR arc non-trivially orthogonal, under the assumption 
that Rp P) R pq and R^ are small, is monotonically related to the integral of square curvature 
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where again die subscript I indicates partial differentiation with respect to the parameter t 
Thus, we can combine die above discussion into the following result. 
Theorem I (Surface Consistency Theorem): Consider some region of a surface 



|/(^,y): {z>y)e<&(ZE 2 \ 



If the reflectance surface R is isotropic, its normal VR is uniformly distributed and the second partials 
Rppj R Pq ,Rqq are small then the probability of a zero-crossing in (v • V) 2 E for some direction v, with 
associated directional angle a, is a mono tonic function of 
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and of 

' {ptqu — QtPu) 2 



ILL 
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where the integral with respect to t is taken along the parametric curve obtained by slicing the image in 
some direction, and the integral with respect to % is taken overall such slices of the image region 9J>.| 

This theorem has the immediate corollary of specifying die "best" surface to fit through a set of 
known points. 
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Corollary 1,1: The surface f most consistent with a set of known points provided by the visual 
system (e.g. from stereo) is that which is least likely to introduce zero-crossings not present in the primal 
sketch. By the Surface Consistency Theorem, this surface minimizes arclength of the parametric normal 
to the surface along each direction and cuiyature of (he parametric normal along each direction, subject 
to passing through the known points.^ 

Finally, wc would like to obtain a simple expression for this surface. Note that the curvature of a 
curve at a point is simple the rate of change of its tangent at that point. Thus, die curve with minimum 
integral square curvature is one with a tangent as nearly constant as possible, relative to the constraints 
imposed by the known points. Next, to minimize arclenth, we require the curve of constant tangent to 
have as small a tangent as possible. This can be done by find the curve which minimzies 

fp 2 t+<fidt 

which is equivalent to 

/ {p x cos a -f- p y sin a) -f- [q x cos a -j- q y sin a) at. 

Integrating this form over all directions a, and using the fact that p y = q x (since / is assumed to be 
twice continuously differentiate), we obtain 

/2 i 2/2 _i_/2 . 

J xx i J xy « J yy 

As a consequence, we obtain the following result 

Corollary 1.2: The surface f which "best" fits a set of known data in a region % of the image is 
that which passes through the known points and minimizes 
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It is interesting to note that this expression for "best" surface interpolation as the surface which 
minimizes the integral of quadratic variation has been derived elsewhere from an analysis of fitting 
this elastic plates to the known points [Grimson, 1981b, 1982]. 

Three assumptions were made in the statement of the surface consistency theorem. One was 
that die reflectance surface R is isotropic, a second, that its normal VR is uniformly distributed, 
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and a third, "that the second order partial derivatives of R are small. What do these assumptions 
imply about the imaging situation? The assumption of an isotropic reflectance function is minor. 
The assumption of a uniform distribution of the reflectance normal is stronger. Mthough a stronger 
version of the surface consistency theorem can be proved by weakening this assumption, we can 
also justify die assumption by an appeal to the "general position" principle of viewer geometry and 
illuminant geometry. The notion of general position requires that all properties of the observed image 
irradiances be roughly independent of die specific viewer position or illuminant position. In other 
words, a small movement of the viewer or the illuminant should not grossly affect the properties of 
die image irradiances. In terms of die reflectance surface, a slight alteration of the viewer position 
or illuminant position will generally result in a translation (and possibly a scaling) of the reflectance 
surface R relative to die p-q coordinate system. Within the proof of die theorem, die assumption 
of uniform distribution is used to argue that at any point in the image, die reflectance normal VR 
is equally likely to be found in any direction. By applying the principle of general position, we can 
loosely argue that die illuminant is equally likely to be in any of a range of positions and that the 
viewer is equally likely to be in any of a range of positions. As a consequence, the reflectance normal 
corresponding to a point in the image is equally likely to be found in any direction, and hence is 
uniformly distributed. The assumption of small. R pp> R pq and R qh indicates that the theorem may not 
be valid for highly specular surfaces, but die assumption is reasonable for surfaces diat are generally 
matte in nature. 

4.4 The Importance of the Surface Consistency Theorem 

We began diis paper by discussing the problem of creating complete surface representations 
from sparse data such as that provided by stereo, and argued that additional constraints were needed 
to effect such a surface reconstruction. The above corollary suggests that one such constraint has 
been identified. In particular, given a class of surfaces that pass through a set of known points, the 
surface which is most consistent with the imaging information is that which is least likely to introduce 
additional zero-crossings. By the surface consistency dieorem, we see that this surface is defined as 
the surface that minimizes die functional of quadratic variation, or at least minimizes some measure 
which is monotonically related to this functional. In [Crimson, 1982], we combine this fact with 
additional constraints, in order to develop a computational theory of visual surface interpolation. 
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5. Examples 

To illustrate these results, we show how inflections in the surface can cause zero-crossings in the 
convolved image, for different light source positions. Figures 7, 8, 9 and 10 show examples of a one- 
dimensional slice of a developable surface, with a Lambcrtian reflectance function (a perfectly diffuse 
surface which reflects flux uniformly in all directions), and the irradiance values obtained for different 
positions of a point light source. Figure 7 indicates the sample surfaces and the rough positions of 
the light sources for the different examples. Figures 8, 9 and 10 indicate sample surfaces and the 
corresponding irradiance profiles for different positions of the light source, as indicated in Figure 7. 
The positions in the irradiance function which would give rise to a zero-crossing in die convolved 
image are indicated. 

6. Summary 

In this paper, we have seen the need for interpolating the results of the stereo algorithm to 
create complete surface descriptions. Determining which surface to fit to the known stereo data is 
at first sight difficult, since any one of a widely varying family of surfaces could be used. We saw, 
however, that most of these surfaces can be discarded as possible interpolations, since they contain 
rapid fluctuations in surface orientation and these fluctuations should give rise to additional zero- 
crossings, not contained in die stereo data. Such zero-crossings would, of course, be inconsistent with 
the imaging data, and hence such a surface is considered to be unacceptable as a possible interpolation 
of the known data. Thus, we proved that the surface consistency constraint restricts the problem: 
The absence of zero-crossings constrains the possible surface shapes 

To make this constraint precise, we outlined the physics of image formation and outlined a 
derivation of Horn's image irradiance equation. We then showed that this expression accounts for 
the zero-crossings of the Marr-Hildrcth edge detection algorithm in the case of surface boundaries, 
surface markings, texture changes, and so forth. The equations farther suggest that even in regions 
where the physical factors causing the previous types of zero-crossings are constant, there may still be 
zero-crossings, due to a change in surface shape. We used this fact to prove the surface consistency 
theorem which relates the probability of a zero-crossing in the second derivative of an image to the 
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Figure 7. Examples of One-Dimensional Surfaces. The top figure shows one surface and the 
arrows indicate the rough orientations of the light source. The numbers refer to the irradiance 
profiles in Figure 8. The bottom figure shows a second surface, with a set of rough orientations of 
the light source. The numbers refer to the profiles of Figures 9 and 10. (From Grimson [1981b]). 
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Figure 8. Examples of a Surface with Two Inflections. The top figure shows a slice of a surface. 
The bottom three figures indicate irradiance profiles for different positions of the light source. 
Note that in all cases, there are six irradiance inflections. In case 3, the irradiances also undergo a 
self-shadowing, where the irradiance value is zero. The positions in the irradiance function which 
would give rise to a zero-crossing in the convolved image are indicated. (From Crimson [1981b]). 
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Mgure y. Examples or a Surface witii One Inflection. The top figure shows a slice of a surface. 
The bottom three figures indicate irradiance profiles for different positions of the light source. 
The positions in the irradiance function which would give rise to a zero-crossing in the convolved 
image are indicated. (From Crimson [1981b]). 
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Figure 10. Examples of a Surface with One Inflection. The top figure shows a slice of a surface. 
The bottom three figures indicate irradianee profiles for different positions of die light source. Note 
that in case 8, the irradianee profile undergoes a self-shadowing. The positions in die irradianee 
function which would give rise to a zero-crossing in the convolved image are indicated. (From 
Grimson [1981b]). 
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variation in the original surface. The surface consistency constraint can be used to formulate a com- 
putational theory of surface interpolation and a specific algorithm for computing the interpolated 
surface [Grimson, 1982]. 



7. Appendix 

Theorem 1 proved a relationship between any general two-dimensional surface and the prob- 
ability of a zero-crossing occuring in die corresponding convolved intensities. Because the theorem 
dealt with any surface, the relationship was somewhat weak. In this appendix, a set of alternate 
theorems are given which apply in cases where die surface can be considered roughly cylindrical (or 
developable). The proofs of these theorems are included for completeness. 

Theorem 2: Consider a portion of a second different table, developable surface oriented along the y 
axis such tha(f y (x l y) = q~ c } for some constant c. If the following conditions are true: 
L The surface portion contains exactly two inflection points in the x direction, at x\ andxo, 
2. At the points x\ and x% normalized changes in albedo are dominated by normalized changes in 
reflectance, 



V 2 p 
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3. The reflectance mapR does not pass through an extremum in this region of the surface, 

4. The reflectance map R is not constant over this region of the surface, 

5. The albedo p is non-zero, 

then there exists a point x\ < x' < x 2 such that V 2 E(a/) = 0. 
Proof The signum function is defined by: 

' 1 if x > 

sgn{x) = ^—l if x < 

if x = 

From the derivatives of the image equation, 

V 2 E = {V 2 p)R + 2 Px R pPx + p(R ppP l + R pPxx ). 
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At the inflection points x t and x 2 , p x {x { ) = 0. Hence, evaluation of the equation yields 

V 2 E(x { ) = V 2 p{x i )Rfr{z i )) + p{xi)RM*i))pU*i) for * = 1, 2 

Condition (2) implies that the albedo changes are negligible in this region, so that the first term may 
be ignored, 

sgn{V 2 E(xi)) = sg^pix^R^pix^p^Xi)). 
Condition (5) implies that sgn(p) = 1. Observing that 

sgn(xy) — sgn(x) sgn(y) t 

the sign of the convolved intensity function at the surface inflection points is given by 

sgn(V 2 E{xi)) = 5^n(fl p (p(^))p^(^)). 

Condition (3) implies that R p does not change sign in this region of the surface and hence 

sgn(R p (p(xi))) = sgn(R p (p{x 2 ))). 

Note that pa? = at x u x 2 . The fact that there are exactly two inflections on the surface implies that 
p x =£Q over the neighbourhood {x u x 2 ). Thus, 

sgn(p xx {xi)) -£ sgn(p xa! (x2)), 

and thus 

sgn^Eix,)) ^ sgn{V 2 E{x 2 )) . 

The second differentiability of the surface implies that there exists a point x f £ {x h x 2 ) such that 
V 2 E{x!) = . | 

Corollary 2.1: Suppose one is given a set of known depth points at a set of zero-crossings, along a 
developable surface. If the albedo p is non-zero and the reflectance mapR is not constant for this region 
of the surface, then the surface cannot contain two or more inflection points between any pair of adjacent 
zero- crossings. | 
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Theorem 3: Consider a portion of a second differentiate developable surface oriented along the y 
axis such that f y (x 9 y) — c, for some constant c. If the following conditions are true; 

1. The surface contains exactly one inflection point in the x direction at x h and the reflectivity R 
achieves an extremum at the point x 2 , x\ 7^ x^, 

2. At the point x x the normalized changes in albedo are dominated by normalized changes in 
reflectance of the form, 
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and at the point x 2 the normalized changes in albedo are dominated by normalized changes in reflectance 
of the form, 
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5. The reflectance mapR is not constant, 
4. The albedo p is non-zero, 

then there exists a point x\ < x f < x 2 such that V 2 E{x!) = 0. 

Proof As in the proof of the previous theorem, at the point x i} 



8gn(V 2 E(x!)) = sgnfofaixifip^fa)). 



At the point x 2} R p — 0so that 



V'E(x 2 ) = V 2 p{x 2 )R{p{x 2 )) + p(x 2 )R pp (p(x2))pl(x 2 ) . 



Condition (2) then implies that at this point, the normalized albedo changes are dominated by the 
normalized reflectance changes, 

sgn{V 2 E{x 2 )) = sgn(p(x 2 ))sgn(R pp (p(x 2 )))sgn(pl(x 2 )) . 

Condition (4) implies that sgn(p) = 1, so that 

sgn(V 2 E(x 2 )) = sgn(R pp (p{x 2 )))sgn(pl{x 2 )) 
= sgn(R pp {p(x 2 ))) 
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There are two subcases. In the first subcase, p xx (xi) > 0. Since there is only one inflection point 
in the surface, this implies thatp(rci) < p{x 2 ). Then R pp (p(x 2 )) < implies that 

«p(pM>«p(p(^)) = 0. 
Conversely, R pp (p(x 2 )) > implies that 

Rpipfa)) < R P (p{*2)) = o. 

■ In ci flier case, 

sgn(R pp {p(x 2 ))) =£ sg^R^x^p^xx)). . 

In the second subcase, suppose that p xx {x{) < 0. This implies that p(x{) > p(x 2 ). Then 
R pp (p(x 2 )) < implies that 

Rp(p{xi))<R P {p(z2)) = 0. 
Conversely, R pp {p{x 2 )) > implies that 

Rpfafa)) > Rpipfa)) = o. 

In either case, 

sgn(R pp {p{x 2 ))) jk sgnftpipfatyp^ixi)). 

Thus, we see that 

sg^Efa)) ^ sgn{V 2 E{^)) 

and as before the second differentiability of the surface implies that there exists a point x f £ {xi, x 2 ) 
such that V 2 ^') = 0.| 

Corollary 3.1: Suppose one is given a set of known depth points at a set of zero-crossings, along a 
developable surface. If the albedo p is non-zero and the reflectance mapR is not constant for this region 
of the surface, then if the reflectance map R passes through an extremum, the surface cannot contain any 
inflection points between any pair of zero-crossings. | 

Theorem 4; Consider a second differentiable developable surface oriented along the y axis such 
thatf y (x f y) — c, for some constant c. If the following conditions are true: 
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At the point %\, the surface becomes self-shadowing, that isR(xi) = 0, 

The reflectivity R achieves an extremum at the point x^, x\ 7^ X2, 

At the point x% normalized changes in albedo are dominated by normalized changes in reflectance, 



p 


< 


R PP P 2 X 
R 



4. The reflectance mapR is not constant over this region, 

5. The albedo p is non-zero, 

then there exists a point x\ < x f < x 2 such that V 2 E(x f ) = 0. 

Proof. The fact that the surface becomes self-shadowing implies that there is a region of the 
surface, beginning at x h such that R is constantly zero. The fact that there is an extremum in R 
for some other point implies that the intensity function must be concave down in the region of the 
extremum and concave up in the region of self-shadowing. There must be an inflection point in 
between and hence there must be a point x' such that V 2 E{x r ) = 0. | 

Theorem 5; Consider a second differentiahle developable surface oriented along the y axis such 
thatfy(x ) y) = c, for some constant c. If the following conditions are true: 

1. At the point x\, the reflectivity R achieves an inflection point, 

2. There exist points x$ < x\ < x 2 such that reflectance changes dominate albedo changes, 



V 2 p R, 
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3. The first derivative of the surface, p, is monotonic in this region, (Le. it does not achieve an 
extremum), 

4. The reflectance map R is not constant over this region, 

5. The albedo p is non-zero, 

then there exists a point xi < x f < x 2 such that V 2 E(x!) = 0. 

Proof The proof is very similar to die previous ones, except that in this case, by condition (2), 

sgn(V 2 E) = sgn{p)sgn(R pp )sgn{p 2 x ) . • 

at the points xq, x 2 . Then condition (4) implies that 
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sgn{V 2 E) = sgn(R pp ) . 
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Condition (3) implies that 

p[xo) < Pfcl) < pfe) 
or 

pfe) < p(*i) < P{xo) 
In citlicr case, condition (1) then implies that 

sgn(R pp {p(xo))) ^ sgn{R pp (p(x 2 ))) . 

Hence, 

sgn(V 2 E{xo)) ^ ^n( V 2 £(a; 2 )) , 

and as before, the second differentiability of the surface implies that there exists a point z! £ (xq, ar 2 ) 
such that V 2 E{x l ) = 0. | 
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